<!DOCTYPE HTML>
<html>
<head>
  <meta charset="utf-8">
  
  <title>htop.me</title>
  <meta name="author" content="Zhaifg">
  
  
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">

  
  <meta property="og:site_name" content="htop.me"/>

  
    <meta property="og:image" content="undefined"/>
  

  
  
    <link href="/favicon.png" rel="icon">
  
  
  <link rel="stylesheet" href="/css/bootstrap.min.css" media="screen" type="text/css">
  <link rel="stylesheet" href="/css/font-awesome.css" media="screen" type="text/css">
  <link rel="stylesheet" href="/css/style.css" media="screen" type="text/css">
  <link rel="stylesheet" href="/css/highlight.css" media="screen" type="text/css">
  <link rel="stylesheet" href="/css/google-fonts.css" media="screen" type="text/css">
  <!--[if lt IE 9]><script src="//html5shiv.googlecode.com/svn/trunk/html5.js"></script><![endif]-->

  <script src="/js/jquery-2.0.3.min.js"></script>

  <!-- analytics -->
  


<script>
var _hmt = _hmt || [];
(function() {
  var hm = document.createElement("script");
  hm.src = "//hm.baidu.com/hm.js?null";
  var s = document.getElementsByTagName("script")[0]; 
  s.parentNode.insertBefore(hm, s);
})();
</script>


</head>

 <body>  
  <nav id="main-nav" class="navbar navbar-inverse navbar-fixed-top" role="navigation">
    <div class="container">
      <button type="button" class="navbar-header navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
		<span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>
	  <a class="navbar-brand" href="/">htop.me</a>
      <div class="collapse navbar-collapse nav-menu">
		<ul class="nav navbar-nav">
		  
		  <li>
			<a href="/archives" title="All the articles.">
			  <i class="fa fa-archive"></i>Archives
			</a>
		  </li>
		  
		  <li>
			<a href="/Categories" title="All the categories.">
			  <i class="fa fa-folder"></i>Categories
			</a>
		  </li>
		  
		  <li>
			<a href="/Tags" title="All the tags.">
			  <i class="fa fa-tags"></i>Tags
			</a>
		  </li>
		  
		  <li>
			<a href="/about" title="About me.">
			  <i class="fa fa-user"></i>About
			</a>
		  </li>
		  
		</ul>
      </div>
    </div> <!-- container -->
</nav>
<div class="clearfix"></div>

  <div class="container">
  	<div class="content">
    	 <div class="page-header">
  <h1>htop.me</h1>
</div>

<div class="row page">

	
	<div class="col-md-9">
	

		<div class="slogan">
      <i class="fa fa-heart"></i>
      少年易学老难成，一寸光阴不可轻。
</div>    

		<div class="mypage">
		
		<!-- title and entry -->
		
			
	
	<!-- display as entry -->	
		<h3 class="title">
			<div class="date"> 2015-07-10 </div>
			<div class="article-title"><a href="/2015/07/10/MySQL复制/" >MySQL主从复制</a></div>						
		</h3>
	


			<div class="entry">
  <div class="row">
  
	<div class="col-md-12">
	  
	
	  <h2 id="MySQL复制">MySQL复制</h2><p>MySQL复制支持单向，异步复制。通过一台主服务器将更新写入二进制日志文件，并维护文件的一个索引以跟踪日志循环。这些日志可以记录发送到从服务器的更新。当一个从服务器连接主服务器时，它通知主服务器从服务器在日志中读取的最后一次成功更新的位置。从服务器接收从那时起发生的任何更新，然后封锁并等待主服务器通知新的更新。MySQL主从复制是异步进行的。同步需要版本为5.5,使用google提供的插件来实现。</p>
<p>MySQL主从复制操作很灵活既可以实现整个服务的级别的复制，也可以实现对某个库，甚至某个数据库中的指定的某个对象进行复制。</p>
<h2 id="MySQL主从复制应用场景">MySQL主从复制应用场景</h2><ol>
<li><p>提高性能。通过一(多)主多从的部署方案，将涉及数据写的操作放在Master端操作，而将数据读的操作分散到众多的Slave当中。降低了Master的负载，提高数据写入的响应效率；多台从服务器提供读，分担了请求，提高了读的效率。</p>
</li>
<li><p>数据安全。数据复制到Slave节点，即使Master宕机，Slave节点还保存着完整数据。</p>
</li>
<li><p>数据分析。将数据分析，放在slave上。</p>
</li>
</ol>
<h2 id="主从复制的原理">主从复制的原理</h2><p>MySQL的Replication是一个异步的复制过程，从一个MySQL实例(Master)复制到另一台MySQL实例上。在Master和Slave之间复制的整个过程主要由3个线程完成，其中两个线程（SQL线程和IO线程）在Slave端，另外一个线程(IO线程)在Master端。</p>
<p>要实现主从复制，首先要在Master端打开Binary Log功能。因为整个复制过程实际上就是Slave从Master上获取二进制日志，然后在自己身上完全按照产生的顺序一次执行日志中记录的各种操作的过程。</p>
<p>复制的具体过程如下：</p>
<p><img src="http://7xkabv.com1.z0.glb.clouddn.com/htop_mysql_replication.jpg" alt="MySQL主从复制原理图"></p>
<ol>
<li>Slave的IO线程连上Master，并请求日志文件指定位置(或从开始的日志)之后的日志的内容。</li>
<li>Master接收到来自Slave的IO线程请求后，负责复制IO线程根据请求的信息读取指定日志之后的日志信息，返回给Slave端的IO线程。返回信息中除了日志所包含的信息，还包含了包括本次返回的信息在Master端的Binary Log文件的名称和位置。</li>
<li>Slave的IO线程接受到信息后，将日志内容一次写入Slave端的Relay Log文件(mysql-relay-bin.xxxx)的末端，并将读取到的Master端的bin-log的文件和位置记录到master-info文件中，以便在下一次读取时能够清楚地告诉Master，下次从bin-log哪个位置开始往后的日志内容。</li>
<li>Slave的SQL线程检测检测到Relay Log中更新内容后，会马上解析该Log文件中的内容，还原成在Master端真实执行时的可执行的SQL语句，并执行这些SQK语句。实际上Master和Slave执行同样的语句。</li>
</ol>
<h2 id="Binary_Log_记录方式">Binary Log 记录方式</h2><h3 id="Row_Level">Row Level</h3><p>Binary Log会记录成每一行数据被修改的形式，然后在Slave端再对相同的数据进行修改。</p>
<p><strong>优点</strong>：在Row Level模式下，Binnary Log可以不记录执行的Query语句的上下文相关信息，只要记录哪一行修改了，修改成什么样子。Row Level会详细的记录下每一行数据的修改细节，而且不会出现某个特定情况下的存储过程，或Function，以及Trigger的调用和触发无法被正确复制问题。</p>
<p><strong>缺点</strong>：产生大量的日志内容。</p>
<h3 id="Statment_Level">Statment Level</h3><p>每一条会修改的SQL语句都会记录到Master的Binnary中。Slave端在复制的时候，SQL线程会解析成和原来Master端执行过相同的SQL语句，并再次执行。</p>
<p><strong>优点</strong>：首先，解决了Row Level下的缺点，不须要记录每一行的数据变化，减少了Binnary Log日志量，节约了IO成本，提高了性能。</p>
<p><strong>缺点</strong>：由于它是记录的执行语句，为了让这些语句在Slave端也能正确执行。那么它还必须记录每条语句在执行时的一些相关信息，即上下文信息，以保证所有语句在Slave端被执行的时候能够得到和在Master端执行时相同的结果。另外，由于MySQL发展比较快，很多新功能不断加入，使得MySQL复制遇到了不小的挑战，复制时设计的内容岳父在，越容易出bug。在Statement Level下，目前已发现不少的情况下会造成MySQL的复制问题。主要是在修改数据使用了某些特定的函数货功能后，出现，比如：Sleep()函数在有些版本中就不能正确的复制，在存储过程中使用了last_insert_id()函数，可能会使Slave和Master的到不一致的ID，等等。</p>
<h3 id="Mixed_Level">Mixed Level</h3><p>在Mixed模式下，MySQL会根据执行的每一条具体的SQL语句来区分对待记录的日志形式，也就是在Statement和Row之间选择一种。除了MySQL认为通过Statement方式可能造成复制过程中Master和Slave之间产生不一致数据。(如特殊Procedure和Funtion的使用，UUID()函数的使用等特殊情况)时，它会选择ROW的模式来记录变更之外，都会使用Statement方式。</p>
<h2 id="设置主从">设置主从</h2><p>主从设置的主要步骤是</p>
<ol>
<li>修改Master和Slave的my.cnf中的server-id，使之唯一,开启binlog。</li>
<li>若不停Master时，加入全局锁，记录bin-log文件和bin-log文件的位置，全备要同步的数据库；解除全局锁</li>
<li>若可以停库时，停止主库，物理备份所需要同步的数据库。</li>
<li>在Slave端恢复备份的数据。</li>
<li>用<code>change master</code>在Slave建立与master的联系。</li>
<li>启动Slave。</li>
<li>检查Slave的状态。</li>
</ol>
<h3 id="实例">实例</h3><ol>
<li>不停主库的建立主从复制。</li>
</ol>
<p><strong>第一部分 对主库的操作</strong></p>
<ul>
<li>1、修改主库的文件，开启MySQL的<code>bin-log</code>。（一般安装的时候最好先做好，这样可以不停库。）<br>修改主库的配置文件my.cnf：</li>
</ul>
<figure class="highlight applescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">vim  /etc/<span class="keyword">my</span>.cnf</span><br><span class="line"></span><br><span class="line">server-<span class="property">id</span> = <span class="number">1</span>         <span class="comment"># 设置server-id 确保id为唯一，最好用ip地址，后三位</span></span><br><span class="line">bin-<span class="command">log</span>=mysql-bin    <span class="comment"># 设置 bin-log，这地方可以指定为bin-log的目录？</span></span><br><span class="line"></span><br><span class="line">/etc/init.d/mysqld restart</span><br></pre></td></tr></table></figure>
<p>修改完成后，重启mysql</p>
<ul>
<li><p>2、登录主库，添加从库的同步账号。</p>
<figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">mysql –uroot –p</span><br><span class="line"><span class="operator"><span class="keyword">grant</span>   replication   <span class="keyword">slave</span>  <span class="keyword">on</span>  *.*  <span class="keyword">to</span>  ‘rep’@’<span class="number">192.168</span>.0.%’  <span class="keyword">identified</span> <span class="keyword">by</span> ‘passwd’;</span></span><br></pre></td></tr></table></figure>
</li>
<li><p>3、对主库进行锁表操作，禁止更新，（为了防止备份时的数据不一致问题）。并查看bin-log的名字和日志的（position）。</p>
</li>
</ul>
<figure class="highlight asciidoc"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">flush  tables with read lock；</span><br><span class="line">show master status；</span><br><span class="line"></span><br><span class="line"><span class="header">mysql&gt; show master status;</span><br><span class="line">+------------------+----------+--------------+------------------+</span></span><br><span class="line"><span class="header">| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |</span><br><span class="line">+------------------+----------+--------------+------------------+</span></span><br><span class="line"><span class="header">| mysql-bin.000107 | 38874616 |              |                  |</span><br><span class="line">+------------------+----------+--------------+------------------+</span></span><br><span class="line">1 row in set (0.00 sec)</span><br><span class="line"></span><br><span class="line">mysql&gt;</span><br></pre></td></tr></table></figure>
<p>记录这两个值<br><code>mysql-bin.000107</code><br><code>38874616</code><br>取得<code>bin-log</code> 和 <code>position</code><br><figure class="highlight stata"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mysql -uroot -p'yz11235' -<span class="keyword">e</span> <span class="string">"show master status"</span> | awk '&#123;<span class="keyword">if</span> (NR == 2)&#123; <span class="keyword">print</span> <span class="label">$1</span> <span class="string">"\n"</span> <span class="label">$2</span>;&#125;&#125;' &gt; temp.txt</span><br></pre></td></tr></table></figure></p>
<ul>
<li><p>4、对主库进行全备份。</p>
<figure class="highlight 1c"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mysqldump  -h hostname -uroot -p’’ -A -B -F  <span class="string">| gzip &gt;alldb.sql.gz</span></span><br></pre></td></tr></table></figure>
</li>
<li><p>5、导出数据库后，进行对数据库表解锁。</p>
<figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="operator"><span class="keyword">unlock</span> <span class="keyword">tables</span></span></span><br></pre></td></tr></table></figure>
</li>
</ul>
<p><strong>第二部分 从库上操作</strong></p>
<ul>
<li><ol>
<li>在从库上设置my.cnf，设置server-id，和注释bin-log。<figure class="highlight applescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">server-<span class="property">id</span> = <span class="number">2</span></span><br><span class="line"><span class="comment"># bin-log = mysql-bin</span></span><br></pre></td></tr></table></figure>
</li>
</ol>
</li>
</ul>
<p>重启 mysql。</p>
<ul>
<li><p>2、登录从库，并设置从库信息</p>
<figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="operator"><span class="keyword">change</span> <span class="keyword">master</span> <span class="keyword">to</span></span><br><span class="line">master_host=<span class="string">'192.168.0.106'</span>,</span><br><span class="line">master_port=<span class="number">3306</span>,</span><br><span class="line">master_user=<span class="string">'rep'</span>,</span><br><span class="line">master_password=<span class="string">'111111'</span>,</span><br><span class="line">master_log_file=<span class="string">'mysql-bin.000107'</span>,</span><br><span class="line">master_log_pos=<span class="number">38907472</span>;</span></span><br><span class="line">mast_connect_retry=10;</span><br></pre></td></tr></table></figure>
</li>
<li><p>3、启动 从库</p>
</li>
</ul>
<figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="operator"><span class="keyword">start</span> <span class="keyword">slave</span></span></span><br></pre></td></tr></table></figure>
<ul>
<li>4、查看从库状态，<figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="operator"><span class="keyword">show</span> <span class="keyword">slave</span> <span class="keyword">status</span>\G</span></span><br></pre></td></tr></table></figure>
</li>
</ul>
<p>观察：<code>slave_IO_Running : Yes</code><br>      <code>Slave_SQL_Running : Yes</code><br>以上两个为Yes 说明主从已经成功。<br><code>Second_behind_master = 0</code> 这是从库落后主库的秒数。</p>

	
	</div>
	
  
</div>
	<a type="button" href="/2015/07/10/MySQL复制/#more" class="btn btn-default more">阅读此文</a>
</div>

		
			
	
	<!-- display as entry -->	
		<h3 class="title">
			<div class="date"> 2015-07-07 </div>
			<div class="article-title"><a href="/2015/07/07/服务器硬盘测试相关/" >服务器硬盘测试相关</a></div>						
		</h3>
	


			<div class="entry">
  <div class="row">
  
	<div class="col-md-12">
	  
	
	  <h2 id="磁盘阵列吞吐量与IOPS两大瓶颈分析">磁盘阵列吞吐量与IOPS两大瓶颈分析</h2><h3 id="1-、吞吐量">1.、吞吐量</h3><p>吞吐量主要取决于<code>阵列的构架</code>，<code>光纤通道的大小</code>(现在阵列一般都是光纤阵列，至于SCSI这样的SSA阵列，我们不讨论)以及<code>硬盘的个数</code>。阵列的构架与每个阵列不同而不同，他们也都存在内部带宽(类似于pc的系统总线)，不过一般情况下，内部带宽都设计的很充足，不是瓶颈的所在。</p>
<p>光纤通道的影响还是比较大的，如数据仓库环境中，对数据的流量要求很大，而<code>一块2Gb</code>的光纤卡，所77能支撑的最大流量应当是<code>2Gb/8(小B)=250MB/s</code>(大B)的实际流量，当<code>4块光纤卡</code>才能达到<code>1GB/s</code>的实际流量，所以数据仓库环境可以考虑换4Gb的光纤卡。</p>
<p>最后说一下硬盘的限制，这里是最重要的，当前面的瓶颈不再存在的时候，就要看硬盘的个数了，我下面列一下不同的硬盘所能支撑的流量大小：</p>
<p>　　10 K rpm  15 K rpm  ATA</p>
<p>　　——— ——— ———</p>
<p>　　10M/s 13M/s 8M/s</p>
<p>那么，假定一个阵列有120块15K rpm的光纤硬盘，那么硬盘上最大的可以支撑的流量为<code>120*13=1560MB/s</code>，如果是<code>2Gb</code>的光纤卡，可能需要<code>6块才能够</code>，而<code>4Gb的光纤卡，3-4块就够了</code>。</p>
<h3 id="2、IOPS">2、IOPS</h3><p>决定IOPS的主要取决与阵列的算法，<code>cache命中率</code>，以及<code>磁盘个数</code>。阵列的算法因为不同的阵列不同而不同，如我们最近遇到在hds usp上面，可能因为ldev(lun)存在队列或者资源限制，而单个ldev的iops就上不去，所以，在使用这个存储之前，有必要了解这个存储的一些算法规则与限制。</p>
<p>cache的命中率取决于数据的分布，cache size的大小，数据访问的规则，以及cache的算法，如果完整的讨论下来，这里将变得很复杂，可以有一天好讨论了。我这里只强调一个cache的命中率，如果一个阵列，读cache的命中率越高越好，一般表示它可以支持更多的IOPS，为什么这么说呢?这个就与我们下面要讨论的硬盘IOPS有关系了。</p>
<p>　　硬盘的限制，每个物理硬盘能处理的<code>IOPS</code>是有限制的，如</p>
<p>　　10K rpm 15K rpm   ATA</p>
<p>　　——— ——— ———</p>
<p>　　100 150 50</p>
<p>同样，如果一个阵列有120块15K rpm的光纤硬盘，那么，它能撑的最大<code>IOPS为120*150=18000</code>，这个为硬件限制的理论值，如果超过这个值，硬盘的响应可能会变的非常缓慢而不能正常提供业务。</p>
<p>在<code>raid5</code>与<code>raid10</code>上，<code>读iops没有差别</code>，但是，相同的<code>业务写iops</code>，最终落在磁盘上的iops是有差别的，而我们<code>评估的却正是磁盘的IOPS</code>，如果达到了磁盘的限制，性能肯定是上不去了。</p>
<p>那我们假定一个case，业务的<code>iops是10000</code>，<code>读cache命中率是30%</code>，<code>读iops为60%</code>，<code>写iops为40%</code>，<code>磁盘个数为120</code>，那么分别计算在raid5与raid10的情况下，每个磁盘的iops为多少。</p>
<p>　　raid5:</p>
<p>　　单块盘的<code>iops = (10000*(1-0.3)*0.6 + 4 * (10000*0.4))/120</code></p>
<p>　　= (4200 + 16000)/120</p>
<p>　　= 168</p>
<p>这里的<code>10000*(1-0.3)*0.6</code>表示是<code>读的iops</code>，<code>比例是0.6</code>，<code>除掉cache命中</code>，实际只有<code>4200个iops</code>而<code>4 * (10000*0.4)</code> 表示<code>写的iops</code>，因为每一个写，在raid5中，实际发生了<code>4个io，所以写的iops为16000个</code><br>为了考虑raid5在写操作的时候，那2个读操作也可能发生命中，所以更精确的计算为：<br>　　单块盘的<code>iops = (10000*(1-0.3)*0.6 + 2 * (10000*0.4)*(1-0.3) + 2 * (10000*0.4))/120</code></p>
<p>　　= (4200 + 5600 + 8000)/120</p>
<p>　　= 148</p>
<p>计算出来单个盘的<code>iops为148个</code>，基本达到磁盘极限</p>
<p>raid10</p>
<p>　　单块盘的<code>iops = (10000*(1-0.3)*0.6 + 2 * (10000*0.4))/120</code></p>
<p>　　= (4200 + 8000)/120</p>
<p>　　= 102</p>
<p>　　可以看到，因为raid10对于一个写操作，只发生<code>2次io</code>，所以，同样的压力，同样的磁盘，每个盘的<code>iops只有102个</code>，还远远低于磁盘的极限<code>iops</code>。</p>
<p>在一个实际的case中，一个恢复压力很大的standby(这里主要是写，而且是小io的写)，采用了raid5的方案，发现性能很差，通过分析，每个磁盘的iops在高峰时期，快达到200了，导致响应速度巨慢无比。后来改造成raid10，就避免了这个性能问题，每个磁盘的iops降到100左右。</p>
<h3 id="服务器硬盘检测工具：">服务器硬盘检测工具：</h3><ul>
<li>fio</li>
</ul>
<h2 id="fio">fio</h2><p>FIO是测试IOPS的非常好的工具，用来对硬件进行压力测试和验证，支持13种不同的I/O引擎，包括:sync,mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio 等等。</p>
<p>2、FIO的用法：</p>
<p>FIO分顺序读，随机读，顺序写，随机写，混合随机读写模式。</p>
<p>这是一个顺序读的模式：<br><figure class="highlight nix"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">fio <span class="variable">-filename=</span>/dev/sda <span class="variable">-direct=</span><span class="number">1</span> -iodepth <span class="number">1</span> -thread <span class="variable">-rw=</span>read <span class="variable">-ioengine=</span>psync <span class="variable">-bs=</span><span class="number">16</span>k <span class="variable">-size=</span><span class="number">200</span>G <span class="variable">-numjobs=</span><span class="number">30</span> <span class="variable">-runtime=</span><span class="number">1000</span> -group_reporting <span class="variable">-name=</span>mytest</span><br></pre></td></tr></table></figure></p>
<p>说明：<br><code>filename=/dev/sdb1</code>                   测试文件名称，通常选择需要测试的盘的data目录。filename=/var/data01/test_io 测试过程使用的文件，一般指向一个不存在的文件，fio会以随机的数据生成这个文件。也可以直接指向块设备文件。<br><code>direct=1</code>                                     测试过程绕过机器自带的buffer。使测试结果更真实。<br><code>rw=randwrite</code>                            测试随机写的I/O<br><code>rw=randrw</code>                                 测试随机写和读的I/O<br><code>bs=16k</code>                                      单次io的块文件大小为16k<br><code>bsrange=512-2048</code>                      同上，提定数据块的大小范围<br><code>size=5G</code>                                       本次的测试文件大小为5g，以每次4k的io进行测试。<br><code>numjobs=30</code>                               本次的测试线程为30个.<br><code>runtime=1000</code>                             测试时间为1000秒，如果不写则一直将5g文件分4k每次写完为止。<br><code>ioengine=psync</code>                            io引擎使用pync方式<br><code>rwmixwrite=30</code>                            在混合读写的模式下，写占30%<br><code>group_reporting</code>                           关于显示结果的，汇总每个进程的信息。</p>
<p><code>lockmem=1G</code>                                只使用1g内存进行测试。<br><code>zero_buffers</code>                                 用0初始化系统buffer。<br><code>nrfiles=8</code>                                    每个进程生成文件的数量。</p>
<p>所以，随机读：<br><figure class="highlight nix"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">fio <span class="variable">-filename=</span>/dev/sda <span class="variable">-direct=</span><span class="number">1</span> -iodepth <span class="number">1</span> -thread <span class="variable">-rw=</span>randread <span class="variable">-ioengine=</span>psync <span class="variable">-bs=</span><span class="number">16</span>k <span class="variable">-size=</span><span class="number">200</span>G <span class="variable">-numjobs=</span><span class="number">30</span> <span class="variable">-runtime=</span><span class="number">1000</span> -group_reporting <span class="variable">-name=</span>mytest</span><br></pre></td></tr></table></figure></p>
<p>顺序写：<br><figure class="highlight nix"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">fio <span class="variable">-filename=</span>/dev/sda <span class="variable">-direct=</span><span class="number">1</span> -iodepth <span class="number">1</span> -thread <span class="variable">-rw=</span>write <span class="variable">-ioengine=</span>psync <span class="variable">-bs=</span><span class="number">16</span>k <span class="variable">-size=</span><span class="number">200</span>G <span class="variable">-numjobs=</span><span class="number">30</span> <span class="variable">-runtime=</span><span class="number">1000</span> -group_reporting <span class="variable">-name=</span>mytest</span><br></pre></td></tr></table></figure></p>
<p>随机写：<br><figure class="highlight nix"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">fio <span class="variable">-filename=</span>/dev/sda <span class="variable">-direct=</span><span class="number">1</span> -iodepth <span class="number">1</span> -thread <span class="variable">-rw=</span>randwrite <span class="variable">-ioengine=</span>psync <span class="variable">-bs=</span><span class="number">16</span>k <span class="variable">-size=</span><span class="number">200</span>G <span class="variable">-numjobs=</span><span class="number">30</span> <span class="variable">-runtime=</span><span class="number">1000</span> -group_reporting <span class="variable">-name=</span>mytest</span><br></pre></td></tr></table></figure></p>
<p>混合随机读写：<br><figure class="highlight nix"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">fio <span class="variable">-filename=</span>/dev/sda <span class="variable">-direct=</span><span class="number">1</span> -iodepth <span class="number">1</span> -thread <span class="variable">-rw=</span>randrw <span class="variable">-rwmixread=</span><span class="number">70</span> <span class="variable">-ioengine=</span>psync <span class="variable">-bs=</span><span class="number">16</span>k <span class="variable">-size=</span><span class="number">200</span>G <span class="variable">-numjobs=</span><span class="number">30</span> <span class="variable">-runtime=</span><span class="number">100</span> -group_reporting <span class="variable">-name=</span>mytest <span class="variable">-ioscheduler=</span>noop</span><br></pre></td></tr></table></figure></p>
<h3 id="案例">案例</h3><figure class="highlight nix"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost ~]<span class="comment"># fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30</span></span><br><span class="line"><span class="variable">-runtime=</span><span class="number">100</span> -group_reporting <span class="variable">-name=</span>mytest1</span><br><span class="line"></span><br><span class="line">mytest1: (<span class="variable">g=</span><span class="number">0</span>): <span class="variable">rw=</span>randrw, <span class="variable">bs=</span><span class="number">16</span>K-<span class="number">16</span>K/<span class="number">16</span>K-<span class="number">16</span>K, <span class="variable">ioengine=</span>psync, <span class="variable">iodepth=</span><span class="number">1</span></span><br><span class="line">...</span><br><span class="line">mytest1: (<span class="variable">g=</span><span class="number">0</span>): <span class="variable">rw=</span>randrw, <span class="variable">bs=</span><span class="number">16</span>K-<span class="number">16</span>K/<span class="number">16</span>K-<span class="number">16</span>K, <span class="variable">ioengine=</span>psync, <span class="variable">iodepth=</span><span class="number">1</span></span><br><span class="line">fio <span class="number">2.0</span>.<span class="number">7</span></span><br><span class="line">Starting <span class="number">30</span> threads</span><br><span class="line">Jobs: <span class="number">1</span> (<span class="variable">f=</span><span class="number">1</span>): [________________m_____________] [<span class="number">3.5</span>% done] [<span class="number">6935</span>K/<span class="number">3116</span>K /s] [<span class="number">423</span> /<span class="number">190</span> iops] [eta <span class="number">48</span>m:<span class="number">20</span>s] s]</span><br><span class="line">mytest1: (<span class="variable">groupid=</span><span class="number">0</span>, <span class="variable">jobs=</span><span class="number">30</span>): <span class="variable">err=</span> <span class="number">0</span>: <span class="variable">pid=</span><span class="number">23802</span></span><br><span class="line">**read** : <span class="variable">io=</span><span class="number">1853.4</span>MB, **<span class="variable">bw=</span><span class="number">18967</span>KB/s**, **<span class="variable">iops=</span><span class="number">1185</span>** , <span class="variable">runt=</span><span class="number">100058</span>msec</span><br><span class="line">clat (usec): <span class="variable">min=</span><span class="number">60</span> , <span class="variable">max=</span><span class="number">871116</span> , <span class="variable">avg=</span><span class="number">25227.91</span>, <span class="variable">stdev=</span><span class="number">31653.46</span></span><br><span class="line">lat (usec): <span class="variable">min=</span><span class="number">60</span> , <span class="variable">max=</span><span class="number">871117</span> , <span class="variable">avg=</span><span class="number">25228.08</span>, <span class="variable">stdev=</span><span class="number">31653.46</span></span><br><span class="line">clat percentiles (msec):</span><br><span class="line">| <span class="number">1.00</span><span class="variable">th=</span>[ <span class="number">3</span>], <span class="number">5.00</span><span class="variable">th=</span>[ <span class="number">5</span>], <span class="number">10.00</span><span class="variable">th=</span>[ <span class="number">6</span>], <span class="number">20.00</span><span class="variable">th=</span>[ <span class="number">8</span>],</span><br><span class="line">| <span class="number">30.00</span><span class="variable">th=</span>[ <span class="number">10</span>], <span class="number">40.00</span><span class="variable">th=</span>[ <span class="number">12</span>], <span class="number">50.00</span><span class="variable">th=</span>[ <span class="number">15</span>], <span class="number">60.00</span><span class="variable">th=</span>[ <span class="number">19</span>],</span><br><span class="line">| <span class="number">70.00</span><span class="variable">th=</span>[ <span class="number">26</span>], <span class="number">80.00</span><span class="variable">th=</span>[ <span class="number">37</span>], <span class="number">90.00</span><span class="variable">th=</span>[ <span class="number">57</span>], <span class="number">95.00</span><span class="variable">th=</span>[ <span class="number">79</span>],</span><br><span class="line">| <span class="number">99.00</span><span class="variable">th=</span>[ <span class="number">151</span>], <span class="number">99.50</span><span class="variable">th=</span>[ <span class="number">202</span>], <span class="number">99.90</span><span class="variable">th=</span>[ <span class="number">338</span>], <span class="number">99.95</span><span class="variable">th=</span>[ <span class="number">383</span>],</span><br><span class="line">| <span class="number">99.99</span><span class="variable">th=</span>[ <span class="number">523</span>]</span><br><span class="line">bw (KB/s) : <span class="variable">min=</span> <span class="number">26</span>, <span class="variable">max=</span> <span class="number">1944</span>, <span class="variable">per=</span><span class="number">3.36</span>%, <span class="variable">avg=</span><span class="number">636.84</span>, <span class="variable">stdev=</span><span class="number">189.15</span></span><br><span class="line">**write**: <span class="variable">io=</span><span class="number">803600</span>KB, **<span class="variable">bw=</span><span class="number">8031.4</span>KB/s**, **<span class="variable">iops=</span><span class="number">501</span>** , <span class="variable">runt=</span><span class="number">100058</span>msec</span><br><span class="line">clat (usec): <span class="variable">min=</span><span class="number">52</span> , <span class="variable">max=</span><span class="number">9302</span> , <span class="variable">avg=</span><span class="number">146.25</span>, <span class="variable">stdev=</span><span class="number">299.17</span></span><br><span class="line">lat (usec): <span class="variable">min=</span><span class="number">52</span> , <span class="variable">max=</span><span class="number">9303</span> , <span class="variable">avg=</span><span class="number">147.19</span>, <span class="variable">stdev=</span><span class="number">299.17</span></span><br><span class="line">clat percentiles (usec):</span><br><span class="line">| <span class="number">1.00</span><span class="variable">th=</span>[ <span class="number">62</span>], <span class="number">5.00</span><span class="variable">th=</span>[ <span class="number">65</span>], <span class="number">10.00</span><span class="variable">th=</span>[ <span class="number">68</span>], <span class="number">20.00</span><span class="variable">th=</span>[ <span class="number">74</span>],</span><br><span class="line">| <span class="number">30.00</span><span class="variable">th=</span>[ <span class="number">84</span>], <span class="number">40.00</span><span class="variable">th=</span>[ <span class="number">87</span>], <span class="number">50.00</span><span class="variable">th=</span>[ <span class="number">89</span>], <span class="number">60.00</span><span class="variable">th=</span>[ <span class="number">90</span>],</span><br><span class="line">| <span class="number">70.00</span><span class="variable">th=</span>[ <span class="number">92</span>], <span class="number">80.00</span><span class="variable">th=</span>[ <span class="number">97</span>], <span class="number">90.00</span><span class="variable">th=</span>[ <span class="number">120</span>], <span class="number">95.00</span><span class="variable">th=</span>[ <span class="number">370</span>],</span><br><span class="line">| <span class="number">99.00</span><span class="variable">th=</span>[ <span class="number">1688</span>], <span class="number">99.50</span><span class="variable">th=</span>[ <span class="number">2128</span>], <span class="number">99.90</span><span class="variable">th=</span>[ <span class="number">3088</span>], <span class="number">99.95</span><span class="variable">th=</span>[ <span class="number">3696</span>],</span><br><span class="line">| <span class="number">99.99</span><span class="variable">th=</span>[ <span class="number">5216</span>]</span><br><span class="line">bw (KB/s) : <span class="variable">min=</span> <span class="number">20</span>, <span class="variable">max=</span> <span class="number">1117</span>, <span class="variable">per=</span><span class="number">3.37</span>%, <span class="variable">avg=</span><span class="number">270.27</span>, <span class="variable">stdev=</span><span class="number">133.27</span></span><br><span class="line">lat (usec) : <span class="number">100</span>=<span class="number">24.32</span>%, <span class="number">250</span>=<span class="number">3.83</span>%, <span class="number">500</span>=<span class="number">0.33</span>%, <span class="number">750</span>=<span class="number">0.28</span>%, <span class="number">1000</span>=<span class="number">0.27</span>%</span><br><span class="line">lat (msec) : <span class="number">2</span>=<span class="number">0.64</span>%, <span class="number">4</span>=<span class="number">3.08</span>%, <span class="number">10</span>=<span class="number">20.67</span>%, <span class="number">20</span>=<span class="number">19.90</span>%, <span class="number">50</span>=<span class="number">17.91</span>%</span><br><span class="line">lat (msec) : <span class="number">100</span>=<span class="number">6.87</span>%, <span class="number">250</span>=<span class="number">1.70</span>%, <span class="number">500</span>=<span class="number">0.19</span>%, <span class="number">750</span>=<span class="number">0.01</span>%, <span class="number">1000</span>=<span class="number">0.01</span>%</span><br><span class="line">cpu : <span class="variable">usr=</span><span class="number">1.70</span>%, <span class="variable">sys=</span><span class="number">2.41</span>%, <span class="variable">ctx=</span><span class="number">5237835</span>, <span class="variable">majf=</span><span class="number">0</span>, <span class="variable">minf=</span><span class="number">6344162</span></span><br><span class="line">IO depths : <span class="number">1</span>=<span class="number">100.0</span>%, <span class="number">2</span>=<span class="number">0.0</span>%, <span class="number">4</span>=<span class="number">0.0</span>%, <span class="number">8</span>=<span class="number">0.0</span>%, <span class="number">16</span>=<span class="number">0.0</span>%, <span class="number">32</span>=<span class="number">0.0</span>%, &gt;=<span class="number">64</span>=<span class="number">0.0</span>%</span><br><span class="line">submit : <span class="number">0</span>=<span class="number">0.0</span>%, <span class="number">4</span>=<span class="number">100.0</span>%, <span class="number">8</span>=<span class="number">0.0</span>%, <span class="number">16</span>=<span class="number">0.0</span>%, <span class="number">32</span>=<span class="number">0.0</span>%, <span class="number">64</span>=<span class="number">0.0</span>%, &gt;=<span class="number">64</span>=<span class="number">0.0</span>%</span><br><span class="line">complete : <span class="number">0</span>=<span class="number">0.0</span>%, <span class="number">4</span>=<span class="number">100.0</span>%, <span class="number">8</span>=<span class="number">0.0</span>%, <span class="number">16</span>=<span class="number">0.0</span>%, <span class="number">32</span>=<span class="number">0.0</span>%, <span class="number">64</span>=<span class="number">0.0</span>%, &gt;=<span class="number">64</span>=<span class="number">0.0</span>%</span><br><span class="line">issued : <span class="variable">total=</span><span class="variable">r=</span><span class="number">118612</span>/<span class="variable">w=</span><span class="number">50225</span>/<span class="variable">d=</span><span class="number">0</span>, <span class="variable">short=</span><span class="variable">r=</span><span class="number">0</span>/<span class="variable">w=</span><span class="number">0</span>/<span class="variable">d=</span><span class="number">0</span></span><br><span class="line">Run status group <span class="number">0</span> (all jobs):</span><br><span class="line">READ: <span class="variable">io=</span><span class="number">1853.4</span>MB, <span class="variable">aggrb=</span><span class="number">18966</span>KB/s, <span class="variable">minb=</span><span class="number">18966</span>KB/s, <span class="variable">maxb=</span><span class="number">18966</span>KB/s, <span class="variable">mint=</span><span class="number">100058</span>msec, <span class="variable">maxt=</span><span class="number">100058</span>msec</span><br><span class="line">WRITE: <span class="variable">io=</span><span class="number">803600</span>KB, <span class="variable">aggrb=</span><span class="number">8031</span>KB/s, <span class="variable">minb=</span><span class="number">8031</span>KB/s, <span class="variable">maxb=</span><span class="number">8031</span>KB/s, <span class="variable">mint=</span><span class="number">100058</span>msec, <span class="variable">maxt=</span><span class="number">100058</span>msec</span><br><span class="line">Disk stats (read/write):</span><br><span class="line">sdb: <span class="variable">ios=</span><span class="number">118610</span>/<span class="number">50224</span>, <span class="variable">merge=</span><span class="number">0</span>/<span class="number">0</span>, <span class="variable">ticks=</span><span class="number">2991317</span>/<span class="number">6860</span>, <span class="variable">in_queue=</span><span class="number">2998169</span>, <span class="variable">util=</span><span class="number">99.77</span>%</span><br></pre></td></tr></table></figure>
<p>主要查看以上<strong> </strong>包含部分的iops(read/write)</p>
<p><a href="http://nolinux.blog.51cto.com/4824967/1662942" target="_blank" rel="external">案例1</a></p>
<h3 id="参考：">参考：</h3><p><a href="http://blog.csdn.net/yuesichiu/article/details/8722417" target="_blank" rel="external">fio详细介绍</a></p>

	
	</div>
	
  
</div>
	<a type="button" href="/2015/07/07/服务器硬盘测试相关/#more" class="btn btn-default more">阅读此文</a>
</div>

		
			
	
	<!-- display as entry -->	
		<h3 class="title">
			<div class="date"> 2015-07-07 </div>
			<div class="article-title"><a href="/2015/07/07/自动化运维/" >Python 多线程</a></div>						
		</h3>
	


			<div class="entry">
  <div class="row">
  
	<div class="col-md-12">
	  
	
	  
	
	</div>
	
  
</div>
	<a type="button" href="/2015/07/07/自动化运维/#more" class="btn btn-default more">阅读此文</a>
</div>

		

		</div>

		<!-- pagination -->
		<div>
  		<center>
		<div class="pagination">
<ul class="pagination">
	 
</ul>
</div>

  		</center>
		</div>

		
		
	</div> <!-- col-md-9 -->

	
		<div class="col-md-3">
	<div id="sidebar">
	
			<div class="form-group has-success has-feedback">
  <form action="//google.com/search" method="get" accept-charset="utf-8" >
    <input type="search" name="q" results="0" placeholder="搜索" class="form-control">
    <input type="hidden" name="q" value="site:htop.me">	
  </form>
</div>

		
			
	<div class="widget">
		<h4>分类</h4>
		<ul class="tag_box inline list-unstyled">
		
			<li><a href="/categories/Linux/">Linux<span>1</span></a></li>
		
			<li><a href="/categories/MySQL/">MySQL<span>1</span></a></li>
		
			<li><a href="/categories/Python/">Python<span>1</span></a></li>
		
		</ul>
	</div>

		
			
	<div class="widget">
		<h4>标签云</h4>
		<ul class="tag_box inline list-unstyled">		
		
			<li><a href="/tags/多线程/">多线程<span>1</span></a></li>
		
			<li><a href="/tags/Python/">Python<span>1</span></a></li>
		
			<li><a href="/tags/iops/">iops<span>1</span></a></li>
		
			<li><a href="/tags/MySQL/">MySQL<span>1</span></a></li>
		
			<li><a href="/tags/fio/">fio<span>1</span></a></li>
		
		 
		</ul>
	</div>


		
			
<div class="widget">
  <h4>最新文章</h4>
  <ul class="entry list-unstyled">
    
      <li>
        <a href="/2015/07/10/MySQL复制/" ><i class="fa fa-file-o"></i>MySQL主从复制...</a>
      </li>
    
      <li>
        <a href="/2015/07/07/服务器硬盘测试相关/" ><i class="fa fa-file-o"></i>服务器硬盘测试相关...</a>
      </li>
    
      <li>
        <a href="/2015/07/07/自动化运维/" ><i class="fa fa-file-o"></i>Python 多线程...</a>
      </li>
    
  </ul>
</div>

		
			
<div class="widget">
	<h4>链接</h4>
	<ul class="blogroll list-unstyled">
	
		<li><i class="fa fa-github"></i><a href="https://github.com/jenguo/" title="My Github." target="_blank"]);">My Github</a></li>
	
	</ul>
</div>


		
	</div> <!-- sidebar -->
</div> <!-- col-md-3 -->

	
	
</div> <!-- row-fluid -->
	</div>
  </div>
  <div class="container-narrow">
  <footer> <p>
  &copy; 2015 Zhaifg
  
      with help from <a href="http://hexo.io/" target="_blank">Hexo</a> and <a href="http://getbootstrap.com/" target="_blank">Twitter Bootstrap</a>. Theme by <a href="http://github.com/wzpan/hexo-theme-freemind/">Freemind</a>.    
</p> </footer>
</div> <!-- container-narrow -->
  
<a id="gotop" href="#">   
  <span>▲</span> 
</a>

<script src="/js/jquery.imagesloaded.min.js"></script>
<script src="/js/gallery.js"></script>
<script src="/js/bootstrap.min.js"></script>
<script src="/js/main.js"></script>


<script type="text/javascript">
  var duoshuoQuery = { short_name: 'jenguo' };
  (function() {
    var ds = document.createElement('script');
    ds.type = 'text/javascript';
    ds.async = true;
    ds.src = 'http://static.duoshuo.com/embed.js';
    ds.charset = 'UTF-8';
    (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(ds);
  })();
</script>


<link rel="stylesheet" href="/fancybox/jquery.fancybox.css" media="screen" type="text/css">
<script src="/fancybox/jquery.fancybox.pack.js"></script>
<script type="text/javascript">
(function($){
  $('.fancybox').fancybox();
})(jQuery);
</script>


</body>
   </html>
